Global input device for multiple computer-controlled medical systems

ABSTRACT

An integrated control and display system for operating a plurality of separate computer systems each having a visual display, includes at least one cursor control device for seamlessly moving a cursor among the visual displays of the separate computer systems, the cursor control device operating to control the computer system associated with the display on which the cursor appears.

BACKGROUND

This invention relates to a consolidated user interface systems andmethods and in particular, to an interface for operating multiplemedical systems.

Many of today's most advanced and effective diagnostic and therapeuticprocedures involve the coordinated use of multiple separate medicalsystems. Each of these separate medical systems may include its ownvisual display and its own input device. As a result, the visualdisplays and input devices can take up valuable space in the procedureroom, as they must be placed in position for convenient access and use.These multiple displays and input device also pose a risk of confusionamong the various displays and input devices.

SUMMARY

Generally, the present invention relates to the seamless,workflow-driven operation of multiple separate medical systems.Embodiments of the present invention provide systems and methods for theoperation of two or more medical systems, each having a visual displayand standard input devices.

Generally a preferred embodiment of a system for controlling multipleseparate medical systems comprises a display with an active displayportion and an inactive display portion, an input device, and acontroller for selecting one of the medical systems whose visual displayis displayed on the active region of the display and which is controlledby the input device. For the purpose of this disclosure, “active” meansthat the corresponding medical system or remote computer has beenselected for both inputs and outputs. The controller selects the medicalsystem at least in part according to a predetermined routine based uponthe procedure being performed. Alternatively or additionally, thecontroller selects the medical system at least in part based uponinformation from at least one of the medical system. Alternatively oradditionally, the controller selects the medical system at least in partbased upon user selection.

Generally a preferred embodiment of a method for controlling multipleseparate medical systems comprises displaying the visual display outputof one of the medical systems on a dedicated active display area, anddisplaying the visual display output of at least one non-selectedmedical systems on an inactive display area; communicating commands fromat least one input device to the medical system whose visual displayoutput is being displayed on the active area, and actively switching themedical system whose visual display output is displayed on the activearea. The medical system whose visual display output is displayed on theactive display area is selected at least in part according to apredetermined order based upon the type of procedure being performedwith the medical systems. Alternatively or additionally, the medicalsystem selected to have its visual display output displayed on theactive display is selected at least in part based upon input receivedfrom at least some of the medical systems. Alternatively or additionallythe medical system selected to have its visual display output displayedon the active display is selected at least in part based upon userinput.

In another preferred embodiment, multiple active displays of multiplemedical systems are provided on the composite display, and the movementof a user input device such as a computer mouse is seamlessly tracked bya processor and actively assigned to the appropriate medical systembased on the location of the mouse within the composite display.

Embodiments of this invention provide systems and methods facilitatingthe operation of multiple separate medical systems, simplifying theprocedure control site and reducing the risk of confusion.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view of a preferred embodiment of a system forcontrolling multiple medical devices in accordance with the principlesof this invention;

FIG. 2 is an elevation view of an example display of a system forcontrolling multiple medical devices in accordance with the principlesof this invention;

FIG. 3 is a schematic view of a preferred embodiment of a consolidateduser interface for performing multistep medical procedure utilizing aplurality of separate medical systems;

FIGS. 4A and 4B are schematic diagrams illustrating one possible methodof highlighting the display from a separate medical system on thecomposite display;

FIGS. 5A and 5B are schematic diagrams illustrating one possible methodof highlighting the display from a separate medical system on thecomposite display;

FIGS. 6A and 6B are schematic diagrams illustrating one possible methodof highlighting the display from a separate medical system on thecomposite display;

FIGS. 7A and 7B are schematic diagrams illustrating one possible methodof highlighting the display from a separate medical system on thecomposite display;

FIGS. 8A and 8B are schematic diagrams illustrating one possible methodof highlighting the display from a separate medical system on thecomposite display;

FIGS. 9A, 9B and 9C are schematic diagrams illustrating possibleimplementations of the prompts that are displayed on the compositedisplay;

FIG. 10A is a schematic diagram of a display showing control buttons forscrolling through prompts through the workflow;

FIG. 10B is a schematic diagram of a display showing an example of thechange of appearance of the cursor;

FIG. 11 is a perspective view of one implementation of variousembodiments of the systems of this invention;

FIG. 12 is a view of a composite display illustrating thereconfiguration of the display in accordance with the workflow;

FIG. 13 is a view of a composite display illustrating thereconfiguration of the display in accordance with the workflow;

FIG. 14 is a view of a composite display illustrating thereconfiguration of the display in accordance with the workflow;

FIG. 15 is a schematic diagram of the architecture of one possiblesystem implementing various embodiments of this invention;

FIG. 16 is a schematic diagram of some of the components of a possiblesystem implementing the various embodiments of this invention;

FIGS. 17A and 17B are views of the composite display, illustrating theapparent movement of the a single cursor across the composite display;and

FIGS. 18A and 18B are views of the composite display, illustrating theuse of a single input device to control multiple systems depending uponthe positioning of the cursor.

FIG. 19A and 19B show a diagram of a number of components and systemsnetworked via a USB controller.

Corresponding reference numerals indicate corresponding parts throughoutthe several views of the drawings.

DETAILED DESCRIPTION

A preferred embodiment of a system for controlling multiple medicalsystems is indicated generally as 20 in FIG. 1. As shown in FIG. 1, thesystem 20 is adapted for use in controlling two or more separate medicalsystems. For example as shown in FIG. 1 the system 20 is adapted forcontrolling three separate medical systems, a remote navigation system22, a localization system 24, and an X-ray imaging system 26. While thepreferred embodiment is described in the context of three systems 22,24, and 26, the invention is not so limited, and can be used with anynumber of medical systems, and furthermore can be used with additionalor different medical systems such as an Ultrasound imaging system or anECG recording system. Each of these separate medical systems preferablygenerates visual display data, and preferably also is adapted to receiveinput commands from an input device. Most of these separate medicalsystems are PC or computer based, with a conventional monitor, and aconventional input device such a keyboard and mouse, track ball, joystick, etc.

In this preferred embodiment the system 20 includes a display 28, havingan active display area 30 and a passive display area 32. The active andpassive display areas 30 and 32 can be dedicated portions of a singledisplay device, or they can be dynamically determined portions of asingle display device, or they can be portions of separate displaydevices, which are preferably integrated together.

In this preferred embodiment the system 20 also includes at least oneinput device, such as a keyboard 34 and/or a mouse 36. The system 20 canalso include additional or different input devices such as track balls,joysticks, haptic devices, touch screens, etc.

In this preferred embodiment the system 20 also includes a controller38, which in this preferred embodiment includes a computer, such as PC40. The computer is programmed to display the visual display data from aselected one of the medical systems (e.g. one of systems 22, 24, and 26)on the active area 30 of the display 28. The computer is also programmedto provide control signals from the input devices (e.g. keyboard 34 andmouse 36) to the selected medical system, so that the input devicescontrol the medical system whose visual display data is displayed on theactive display area 30 of the display 28. This helps eliminate confusionin operating the separate medical systems in the course of a procedure.

In this preferred embodiment the computer is preferably programmed tointelligently select the medical system based at least in part upon apredetermined routine for the medical procedure being performed. Thus,for example, some identification of the medical procedure beingperformed is provided to the computer, and the computer then selects themedical system to display in the active display area 30 of the display28. For example this selection can be made based on the time since thestart of the procedure, based upon indications of completion of variousphases of the procedure either obtained from the various medical systemsor from the user. For example after a particular action or command usingone medical system, the computer may be programmed to automaticallyselect another medical system. Alternatively or in addition, thecomputer may select the medical system at least in part based uponinformation received from the various medical systems. For example ifthe computer receives an alarm from a particular system, it mightautomatically select that system to facilitate the user's response tothe alarm. Alternatively or in addition, the computer may select themedical system at least in part based upon information received from theuser, including an express instruction to switch to a different medicalsystem.

In particular the computer can be programmed to select the medicalsystem based at least in part on a predetermined routine for a givenmedical procedure, and can take into account signals, includingemergency signals, from the various medical systems being used, as wellas inputs from the user reflecting the starting or completion of variousphases or steps of the procedure, and specific requests to select aspecific medical system.

As shown in FIG. 2, the active display portion 30 of the display 20 ispreferably more prominent, for example in size and in location, than theinactive display portions 32. Thus it is immediately apparent whatsystem is being actively displayed, and what system will be controlledby the use of the input devices (e.g. keyboard 34 and mouse 36). Whenthe computer automatically changes selection of the medical system,there is preferably some sort of indication prior to the switch, andpreferably some visible and/or audible signal at the switch.Furthermore, in one embodiment the input devices (e.g. keyboard 34 andmouse 36) may be temporarily locked out just prior to the switch untiljust after the switch, to reduce the risk that commands intended for onesystem are communicated to a different system.

The computer 40 can be programmed to prompt the user through apredetermined sequence of steps to conduct the procedure, the programautomatically changing the selected medical system as the user respondsto the on screen prompts, thereby automatically passing control ofappropriate medical system to the keyboard 34 and mouse 36 (or otherinput devices) at the appropriate times. A portion of the display 28 canbe dedicated to displaying prompts, or the prompts can be dynamicallyallocated.

For example a Clinical Workflow Manager program can be run on thecomputer 40 to direct the user to perform the steps necessary to performthe procedure, such as setting up an imaging, localization, and magneticnavigation systems for performing a procedure with magneticallynavigated devices, and to make the systems required to perform theprocedure accessible to the user via the system 20, and in particularthe display 28 and the keyboard 34 and mouse 36.

The Clinical Workflow Manager Program might begin with a prompt on thedisplay 28 to “Import Pre-Op image” or similar designation. When theuser selects this, a Pre-Op window is opened, and a dialog box enablesselection of the appropriate image. The image is loaded into the Pre-Opwindow. A prompt on the display 28 then prompts the user to move to thenext step of acquiring a first X-ray image for registration.

During the acquisition of the first X-ray image for registration theuser takes an X-ray image, and the active portion 30 of the display 28displays the same image as the X-ray monitor. The x-ray system controls(zoom level, frame rate, window width, window level, etc.) are availableon this “Live X-ray” display using the keyboard 34, mouse 36, or otherinput devices. As the user moves a cursor across this active portion 30of the display 28 with the mouse 36, mouse movements are suitably scaledby the processor and fed to the x-ray system computer. Mouse clicks arealso fed to the x-ray system computer. Thus X-ray system controlsettings may be changed if desired. When the user presses the X-ray footswitch to acquire an X-ray image, the user can directly observe theimage on the active portion 30 of the display. The image isautomatically stored to a separate “Reference X-ray” window when themouse cursor is in the appropriate display area on the composite displaycorresponding to the X-ray system.

The Clinical Workflow Manager can then display a prompt on the display28 to prompt the user to move to the next step of acquiring a secondX-ray image for registration, the second x-ray image possibly beingacquired at a different angle with respect to the patient. The x-raysystem controls are accessible to the user from the “Live X-ray” window.When the second image is also selected, the Clinical Workflow Managercan then display a prompt on the display 28 to move to the next step ofmarking landmarks on the pair of X-ray images. A pair of x-ray images isdisplayed to the user on the active portion 30 of the display. After theuser marks a set of corresponding points on the two images (thusdefining a set of 3D points through epipolar geometry) using thekeyboard 34 and mouse 36, the Clinical Workflow Manager can then displaya prompt on the display 28 to select corresponding points on the Pre-Opimage. The Clinical Workflow Manager can then display the Pre-Op imagein the active portion 30 of the display 28. The user may rotate andtranslate the Pre-Op image as desired and select a set of landmarks. Thelandmarks can be matched with the corresponding set of X-ray landmarksfrom the previous step and the system effects a registration between thePre-Op image and X-ray coordinates.

The Clinical Workflow Manager can then display a prompt on the display28 to start Auto Map moves (apply preset field sequence, etc.) andswitch the localization/mapping system display to the active portion 30of the display 28. A sequence of moves is made and the user may move themouse 36 to select and “freeze” points as one would typically do with alocalization system in order to create an anatomical/ECG map. Mousemoves made on (or with reference to) the action portion 30 of thedisplay 28 are suitably scaled and fed to the localization systemcomputer, so that localization user interface tools can be accessed andused via system 20.

The mapping process continues until a complete or suitable anatomicalmap is obtained. The Pre-Op image may at any time be displayed as wellupon user selection, so that anatomical targets can be selected fromthis image as locations to drive the catheter to (in order to furtherrefine a map, for instance). This sequence outlines a mapping procedurethat can be carried out with the system 20 coordinating separate imagingand localization and navigation systems. Although this sequencedemonstrates an implementation with user selected screen displays on thesystem 20, other variations can consist entirely of an automatedselection of displays (for instance: (i) as soon as an X-ray istransferred, the computer picks the next window/system to display, or(ii) instead of prompting the user to select the next display as in theabove, the processor directly switches the display as appropriate), or acombination of automatically selected and user selected displays atvarious steps. Likewise, the displays of various systems such asLocalization system, ECG system, blood pressure monitoring system,X-ray, Ultrasound or other imaging system, remote navigation system, andso on can be displayed on the Consolidated UI in various sequences asappropriate for the procedure.

Operation

In the preferred embodiment shown in FIG. 1, a single keyboard 34 andmouse 36 is attached to the controlling PC 40. This PC 40 produces abackground image output to a video output that is combined with otherclinical video streams by a Video scaler/multiplexer 42. The scalermultiplexer does the image processing necessary to produce an integrateddisplay. An example of an available video scaler/multiplexer is theTVOne C2-7100 w/ RS-232 control. It provides two programmable PIP(Picture in picture) windows overlaid on a main window and acceptsmultiple DVI and VGA inputs with an HDTV output.

A software program on the controlling PC 40 monitors the keyboard 34 andmouse 36 and passes information on as necessary to the remotelyswitchable kvm 44 The remotely switchable kvm 44 routes keystrokes tothe appropriate medical system computer as commanded by the controllingPC. An example of a remote KVM that could do this is the Black BoxKV3108SA-R5, 8 port controllable KVM

The medical systems (e.g. systems 22, 24, and 26) can have their ownmouse/keyboards that serve as backup controls. These could be attachedvia keyboard splitter and mouse splitter cables.

Of course the system 20 does not have to use a PC 40, and instead anydevice capable of producing a video image could be substituted. Thecomputer would need to at least have a microprocessor with somesoftware. Any clinical video producing device that could be controlledby a keyboard/mouse would be plugged into the KVM. It is possible thatsome medical systems will only produce video display data, and notaccept inputs. These devices could nonetheless be integrated with thesystem 20, and displayed at appropriate times on the active display area30 of the display 28, even though the input devices do not provide anycontrol signals.

A second embodiment of a consolidated user interface in accordance withthe principles of this invention is indicated generally as 100 in FIG.3. The system 100 is adapted for operating a plurality of separatemedical systems in order to conduct a medical procedure. As shown inFIG. 3, the system 100 can be used to coordinate and control multipleseparate PC-based medical systems, a medical navigation system 102 fororienting and/or advancing a medical device in an operating region in asubject, a medical localization system 104 for determining the positionand/or orientation of the medical device in the operating region; and animaging system, such as an x-ray imaging system, for imaging theoperating region. Of course, additional or different PC-based medicalsystems could be controlled by the system 100. Furthermore, the separatemedical systems do not have to be PC-based, and could be controlled bysome other type of computer. However each of the separate medicalsystems preferably accepts user inputs via input devices such askeyboards, mice, track balls, etc., and preferably generates anassociated visual display.

Each of the separate systems 102, 104, and 106 are operatively connectedto the CPU 108 of the system 100. This connection can be a dedicatedwire or wireless connection, or connection via a network. The CPU 108 isprogrammed to display the associated displays of at least two of theseparate medical systems on a composite display 110. The compositedisplay 110 is preferably a single display, such an LCD flat paneldisplay, a CRT display, a plasma display, or a projection display.However, the composite display could comprise a plurality of separatedisplays that are associated together in a manner for convenient viewingby users.

As shown in FIG. 3, the composite display 110 has a portion 112 fordisplaying the associated display of the navigation system 102, aportion 114 for displaying the associated display of the imaging system104, and a portion 116 for displaying the associated display of thelocalization system 106. These portions 112, 114, and 116 can bededicated to displaying the associated display of a particular system,or they can be dynamically assigned. The composite display 110 may alsoinclude a portion 118 for prominently displaying the associated displayof a selected separate medical system.

The system 100 also includes input devices, such as a key board 120 andmouse 122 for operating the system and selectively controlling theseparate medical systems. Other input devices can be provided inaddition to, or instead of, the key board 120 and mouse 122.

In this preferred embodiment the CPU is also programmed to sequentiallydisplay prompts 124 for at least some of the steps of a multistepmedical procedure on the composite display 110. These prompts 124 can bea detailed description of the step, or a more abbreviated messagerecognizable to the user. The prompts can include text or symboliccharacters, and are preferably displayed on a dedicated portion orwindow 126 on the composite display.

The system 100 can display all of the prompts 124 simultaneously, or asubset of the prompts (FIGS. 9A, 9B), for example the current prompt,and the immediately preceding and immediately succeeding prompt (FIG.9B), or even just a single prompt (FIG. 9C). As shown in FIG. 9, theuser may be able to scroll through the prompts, for example with aconventional display scroll bar 128, or the system may advance throughthe prompts automatically, for example in response to detecting that astep has been completed, or in response to the user indicating throughuser interface means that a step has been completed. Where more than oneprompt is displayed, the system preferably also provides some indicatorof the current step, for example with a special character, such as anarrow 130, or by displaying the prompt in a different appearance (suchas a difference size, color, or brightness) as shown in FIG. 9B.

The CPU 108 is preferably programmed to automatically change thecomposite display 110 if completing the currently prompted step involvesuse of one of the separate medical systems. This change in the compositedisplay 110 can take many forms. For example, it can be adding thedisplay of the involved computer controlled medical system to thecomposite display as is shown FIGS. 4A and 4B. In FIG. 4A, theassociated display of the localization system is not displayed on thecomposite display. In FIG. 4B, when the prompt 124 calls for the use ofthe localization system, the associated display of the localizationsystem is displayed at 116, to facilitate the use of the localizationsystem.

Alternatively the CPU can be programmed to highlight the display of theinvolved computer controlled medical system if it is already beingdisplayed on the composite display 110. Highlighting the display of theinvolved computer controlled medical system can be accomplished bydisplaying the associated display of the involved computer controlledmedical system in a prominent location, as shown in FIGS. 5A and 5B. InFIG. 5A, the associated displays of the navigation system, the imagingsystem, and the localization system are all displayed on the compositedisplay 110. In FIG. 5B, when the prompt 124 calls for the use of thelocalization system, the associated display of the localization systemis displayed on the prominent portion 118 of the display 110, replacingthe associated display of the navigation system.

Alternatively, highlighting the display of the involved computercontrolled medical system can be accomplished by resizing the associateddisplay of the involved medical system as shown in FIGS. 6A and 6B. InFIG. 6A, the associated displays of the navigation system, the imagingsystem, and the localization system are all displayed on the compositedisplay 110, with the associated display of the navigation system beingenlarged to facilitate the use of the navigation system. In FIG. 6B,when the prompt 124 calls for the use of the localization system, theassociated display of the localization system is enlarged to facilitatethe use of the localization system.

Alternatively, highlighting the display of the involved computercontrolled medical system can be accomplished by applying a border orother indicator to the associated display of the involved medical systemas shown in FIGS. 7A and 7B. In FIG. 7A, the associated displays of thenavigation system, the imaging system, and the localization system areall displayed on the composite display 110, with the associated displayof the navigation system having a prominent border. In FIG. 7B, when theprompt 124 calls for the use of the localization system, a border isdisplayed around the associated display of the localization system.

Alternatively, highlighting the display of the involved computercontrolled medical system can be accomplished by changing the color orbrightness of the involved medical system as shown in FIGS. 8A and 8B.In FIG. 8A, the associated displays of the navigation system, theimaging system, and the localization system are all displayed on thecomposite display 110, In FIG. 7B, when the prompt 124 calls for the useof the localization system, the intensity of the associated display ofthe localization system is increased.

Of course some other manner of highlighting can be used, thehighlighting serving to quickly identify to the user, the display fromthe medical system that is involved in the current step, and/orfacilitating the use of the display from the medical system that isinvolved in the current step.

The system 100 may be able to automatically detect the completion of thestep associated with the prompt of the current step, and automaticallyadvance the prompt. Alternatively, the user can operate a “completed”130 or “next” 132 control (FIG. 10) to cause the system 100 to displaythe prompt for the next step. The prompts can be predetermined for aselected procedure, and for example recalled from memory or storage.Alternatively, the prompts can be dynamically generated based at leastin part from user commands, and/or inputs from the various medicalsystems. Thus, for example the system 100 might present the user withtwo or more alternatives, and the user can select one of thealternatives using selection buttons 134. The system 100 may also makethe appropriate selection for the user, and display just the appropriateselection based in part on user inputs and/or inputs from the variousmedical systems.

In the preferred embodiment the composite display 110 is configured sothat the input devices control a single displayed cursor, and whereverthe displayed cursor is located, the input devices, such as the keyboard and mouse, can be used to operate whatever system is associatedwith the display over which the cursor is positioned. Thus, bymanipulating the input devices the user can move the cursor over thenentire composite display 110, and when the cursor is positioned over theportion displaying the associated display of one of the separate medicalsystems, the input devices can be used to operate that medical system.In this embodiment, more than one of the medical system displays on thecomposite system is active. The assignment of the input device orcomputer mouse position to the appropriate medical system is performedby the controller processor, which scales the mouse cursor movement overthe composite display based on the current absolute mouse position so asto seamlessly tie the mouse to the appropriate medical system display onthe composite display. The controller unit assigns, routes and feeds acorresponding USB output to the appropriate medical system computer. Inthis manner a single display (the composite display 110), and a singleset of input devices (the keyboard and mouse) can be used to control thenavigation system, imaging system, localization/mapping system, and soon, independently of the separate displays and separate individual inputdevices of those systems.

Thus embodiments of the methods and apparatus of this invention providea way for a user to operate a plurality of separate computer systems, bydisplaying the displays from each of a plurality of separate computersystems on a composite display; and operating a cursor control device tomanipulate a cursor on the composite display (including on the displaysof the separate computer systems), and operating the cursor control tooperate the separate computer system whose display corresponds to thelocation of the cursor on the composite display. The cursor manipulatedon the composite display may change appearance when it is positionedover at least a subset of the displays of the separate computer systems.

In accordance with another embodiment of this invention, a control canbe provided for operating a plurality of separate computer systems. Thecontrol comprises a composite display; a video system for receivingdisplay input from each of the plurality of separate computer systemsand displaying it on the composite display; a processor; a distributionsystem connected to the processor and to each of the separate computersystems, and a cursor control device connected to the distributionsystem, the distribution system receiving input from the cursor controldevice and communicating the input to the processor, and communicatingcursor control input generated by the processor to at least one of theseparate computer systems when the cursor on the composite display is onthe display corresponding to that separate computer system.

In accordance with another embodiment of this invention, a method ofoperating a plurality of separate computer systems is provided. Themethod comprises displaying the separate displays from each of theplurality of separate computer systems on a composite display;translating inputs generated by the operation of a cursor control deviceto control signals to an interface computer to move a cursor in theinterstices between the separate displays on the composite screen and tocontrol signals to the separate computer systems to operate the separatecomputer systems, so that the cursor appears to operate substantiallycontinuously across the entire composite display, and when the cursor ispositioned over the separate display of one of the separate computersystems, the cursor control device operates the separate computersystem. In the preferred embodiment the step of translating inputsgenerated by the operation of a cursor control device to control signalsto an interface computer and to control signals to the separate computersystems includes communicating the signals using the USB protocol.

In accordance with another embodiment of this invention, a method ofoperating a plurality of computer systems is provided. The methodcomprises translating inputs generated by the operation of a cursorcontrol device to control signals to a first computer to operate thefirst computer, and generating and transmitting cursor control signalsto operate at least one other computer system to operate the at leastone other computer system in a corresponding manner. In the preferredembodiment the step of generating and transmitting cursor controlsignals to at least one other computer systems comprises generatingcontrol signals that are different from the control signals for thefirst computer. The step of generating and transmitting cursor controlsignals to at least one other computer systems comprises communicatingsignals via USB protocol.

In accordance with another embodiment of this invention, a method ofcontrolling multiple computer systems running programs withcomplimentary actions is provided. The method comprises accepting inputsfrom the operation of a cursor control device and operating a firstcomputer in accordance with the accepted inputs to achieve a desiredaction, and translating the inputs to a second set of inputs to cause atleast one other computer to perform a complimentary action. In thepreferred embodiment, the inputs are translated into USB inputs thatcause the least one other computer to perform the complimentary action.

One method of accomplishing this is mapping one or more mouse actions ona first computer system (for example operating a particular feature)with a corresponding or complementary action (for example operating thesame or a related feature) on at least one other computer system. Whenone of the mapped actions on the first computer system is performed, thesystem can automatically generate artificial mouse commands to at leastone other computer system to cause that computer system to take acorresponding action. For example a medical navigation system can have auser interface in which the display can be oriented in any of severaldirections, by pointing and clicking on the control on the display forthe medical navigation system. The corresponding controls for orientingthe displays from the medical imaging system and the medicallocalization system can be mapped to the appropriate controls on thenavigation system, so that when the user moves a cursor on the displayfor the navigation system and clicks, artificial mouse commands aregenerated for the imaging and localization systems to cause theirrespective cursors to point to a corresponding control and click,without requiring the user to move the cursor and click in each of thesesystems. In this way, by pointing and clicking in the display of justone of the systems, corresponding actions in the other systems can betaken so that all of the displays are oriented in the same manner.Depending upon the mapping, pointing and clicking on a display of one ofthe systems on the composite display can cause the same action or arelated action to be taken by the other systems.

In another embodiment of this invention a method of performing a medicalprocedure using a plurality of medical systems controlled by separatecomputer systems, in which at least some of the computer systems havecomplementary actions, is provided. The method comprises operating acursor control device to cause a first computer system to perform adesired action; and generating a cursor control signal to cause at leastone other computer system to perform a complementary action. The step ofgenerating a cursor control signal preferably includes using a look-uptable to determine the cursor control signal for the at least one othercomputer system that causes a complementary action to the actionperformed by the first computer system. This complementary action can bea corresponding action, so that the systems perform the same action, ora coordinated action, so that the systems operate together when commandsare made on one of the systems. In specific embodiments, the coordinatedactions are at least in part dependent upon the selected interventionworkflow.

In another embodiment of this invention, an integrating control anddisplay system for operating a plurality of separate computer systems isprovided. The integrating control comprises a master display integratingthe separate displays from each of a plurality of separate computersystems; and a cursor control device for manipulating a cursor on themaster display (including on the displays of each of the separatecomputer systems on the master display). The cursor control deviceoperates the system corresponding to the separate display on which itappears on the master display. This allows the user to manipulate acursor over a single composite display using a single control device,and to control any of a plurality of different computer systems whosedisplays are displayed on the composite display.

This is preferably achieved using USB protocol so that the systems canbe connected without the need for any modification to the system or itsprogramming. Thus, in another embodiment of this invention, a method ofoperating a plurality of separate computer systems without altering thehardware or software of the separate computer systems is provided. Themethod comprises: displaying the separate displays of each of theplurality of separate computer systems on a composite display of amaster computer system; and operating the cursor control device of themaster computer system to manipulate a cursor on the master display, andwhen the cursor on the master display overlies the separate display ofone of the separate computer systems, generating cursor control signalsto operate the separate computer system corresponding to the operationof the cursor control device of the master system. The generated cursorcontrol signals are preferably USB cursor control signals for operatingthe separate computer system.

In another embodiment of this invention, a method of controlling aplurality of separate computer systems is provided. The method comprisesdisplaying the display from each of the separate computer systems on anintegrated display; accepting inputs form a cursor control device tomove a cursor control device across the integrated display. The cursorcontrol being operable to operate the separate computer system overwhich the cursor is displayed on the integrated display.

In another embodiment, a system for controlling a plurality of separatecomputer systems via a master computer is provided. The system comprisesa master computer, including a processor, display for displaying adisplay output from the master computer and each of the separatecomputer systems, and a keyboard for entering instructions to the mastercomputer system. The processor of the master computer system isprogrammed to generate control signals for each of the separate computersystems based on the keystrokes on the keyboard, at least some of thekeystrokes generating control signals for less than all of the pluralityof computer systems. The system is preferably configured so that atleast one keystroke on the keyboard generates a control signal for onlyone of the plurality of separate computer signals. This allows a singlekeyboard to be used to control a plurality of computer systems, whilepreserving at least one key for dedicated control of a particularcomputer system, even while the remainder of the keys are controllingother computer systems.

In one embodiment of this invention, a software parser resides on themaster computer and interprets command modifiers to decide onto which ofthe controlled medical system(s) or computer(s) the command isapplicable. The command format is modified in accordance with the parserdesign to un-ambiguously interpret commands, including generic commandssuch as “rotate image.” In one embodiment, the command syntax is suchthat commands are preceded by a letter, or combination of letters, thatuniquely designate control system(s) to be recipient for the issuedcommand. For example, and in the context of FIG. 3, a preceding “N,”“I”, or “L” would be interpreted as designating respectively thenavigation, imaging, or localization system. According to such aspecific syntax, grouping of letters could be interpreted as designatingthe associated controlled systems, i.e. “NI” would send the issuedcommand to both the navigation and the imaging systems. A parserinterpreting keyboard sequence thus can direct a command, group ofcommands (“macro”), to a subset of the controlled systems or computerscomprising one or more system(s) or computer(s). Clearly, specificsyntax to be used in the context of a complex system include uniqueidentifiers such that, in the example above, the leading letters “NI”would be uniquely decoded and interpreted as described. Such anembodiment enables rapid command issuing and is intuitive to a subset ofusers familiar with computer technology.

Alternatively or additionally, the parser functions as a “globalkeyboard.” Specific commands or command shortcuts direct active controlto one of a plurality of controlled systems. That is, upon the usertyping a short keyboard sequence, active control is passed to a specificcontrolled system as determined by the keyboard sequence contents. Inspecific implementations, the keyboard inputs also moves the mastercomputer mouse cursor onto the active window as determined by thecommand parser.

In another embodiment of this invention, a touch screen is provided aspart of the master control system user interface. Buttons on the touchscreen enable selection of the active system among the plurality ofcontrolled systems. Additionally, buttons on the touch screen areprogrammed to activate control command macros consisting of at least onecommand or operation; in specific embodiments, launching such a macroalso redirects the master control system cursor to a location on theassociated system display to press a button or select a menu item.

In another embodiment of this invention, a method of controlling atleast one of a plurality of separate computer systems based upon theoperation of a master computer system is provided. The method comprisesgenerating synthesized USB control signals for at least one of theplurality of separate computer systems, in response to the operation ofthe master computer systems.

One possible implementation of several of the embodiments of theinvention is illustrated as 200 in FIG. 11, in which a work table 202 isshown with a an integrated display 204, a control key board 206, acontrol mouse 208, and a dedicated system controller 210 are shown. Theterms integrated display, consolidated display and composite display areused interchangeably throughout this specification and connote a singlevisual display capable of presenting multiple individual displays withinits viewable display area, wherein each individual display represents avisual display of a separate computer system. The consolidated display204 has a portion 212 for displaying the associated display from amedical navigation system, such as a Stereotaxis magnetic medicalnavigation system, available from Stereotaxis, Inc., St. Louis, MO,although the display could be from some other medical navigation system.The consolidated display 204 has a portion 214 for displaying theassociated display from a medical imaging system, such as a Siemensx-ray system, although the display could be from some other x-raysystem, or from some other type of imaging system altogether, includingMRI and ultrasound. The consolidated display 204 also has a portion 216for displaying the associated display from a medical localizationsystem, such as a Carlo localization system, available from Biosense,although the display could be from some other medical localizationsystem. The consolidated display 204 also has a portion 218 fordisplaying the associated display from an EGG system. The consolidateddisplay 204 can have other portions for displaying the displays fromother separate medical systems, and/or some of the portions 212, 214,216, and 218 could be omitted or replaced.

The consolidated display preferably also includes a window 220 in whichprompts 222, preferably generated by the system, guide the useraccording to a workflow plan. This workflow can be entirelypredetermined based upon the type of procedure, or it can be dynamicallydetermined based in part upon selections by the user and/or inputs fromthe various separate medical systems. In either case the system 200 ispreferably set up to automatically reconfigure the composite display 204for current step in the workflow. The configuration of the compositedisplay 204 for each step can be one that was preset, or it can be onethat was modified and saved by the user.

A single cursor, also referred to as a composite cursor 205 (FIG. 17A,17B), is movable on the consolidated display 204 by using the keyboard206 or mouse 208. The keyboard 206 and mouse 208 are functional tocontrol the separate medical system associated with the portion (e.g.212, 214, 216, and 218) over which the cursor is positioned. Thus theuser can move the cursor from portion to portion by manipulating thekeyboard 206 or mouse 208, and when the cursor is in a particularportion of the consolidated display, use the keyboard 206 and mouse 208to control the associated system. Thus with a single keyboard and mouse,the user can control all of the systems, simply by moving the cursor onthe consolidated display 204. A dedicated controller 210 can be providedwith buttons or other controls that operate one or more of the separatemedical systems regardless of the position of the cursor on theconsolidated display. This allows the user to immediately input commandsto a system irrespective of the position of the cursor. The dedicatedcontroller 210 can operate just one of the systems, such as thenavigation system, or it can have dedicated controls for two or more ofthe systems. The controls can be dedicated to particular functions, orthey can be reprogrammable by the system based upon the context, or bythe user, based upon preference.

Alternatively or additionally, a parsing software running on the mastercontroller enables direct use of the keyboard as a means to specifywhich subset of systems is addressed by a given command. This option forexample is available to the user upon moving the master cursor into themaster computer text output/input window(s). Alternatively the cursor isautomatically moved to the text output/input window upon the userpressing a key or key combinations; such key or key combination beingdefined for example as a key override mode. Alternatively the cursor isautomatically moved to the text output/input window upon the userpressing a combination of mouse buttons or sequence of mouse buttons. Inimplementations where the system includes a touch screen interface,specific buttons on the touch screen are mapped to activate controlcommand macros; such macros can redirect the master control systemcursor to a location on the associated system display to press a buttonor select a menu item.

The operation of the system 200 is illustrated in FIGS. 12-14. In FIG.12, the Carto prompt 222 is highlighted in window 220, and the compositedisplay 204 is automatically configured for using the Carto localizationsystem. The portion 216 displaying the display form the Carto system islarge and centered on the display 204, where it can be easily seen andused by the user. The portions for at least some of the other systems inuse are also active on the display, for example portion 218 displayingthe ECG display, and portion 214 displaying the display from a medicalimaging system. In FIG. 13, the step using the Carto system has beencompleted, and the prompt 222 to use the ECG system is active in window220. The composite display 204 is reorganized, with the potion 218 fordisplaying the ECG display has been enlarged and moved to the center ofthe display 204, while the portion 216 displaying the Carto display hasbeen resized and moved to the corner, and the portion 214 displaying thedisplay from the imaging system remains unchanged. In FIG. 14, the stepusing the Stereotaxis navigation system had been completed, and theprompt 222 to use the navigation system is active in window 220. Thecomposite display 204 is reorganized, with the potion 212 for displayingthe navigation system display has been positioned in the center of thedisplay 204, portion 214 displaying the display from the imaging systemhas been moved to the upper right corner of the display, and the Cartosystem has been moved to the lower right corner of the display.

A possible architecture of the system is shown in FIG. 15. As shown inFIG. 15, a plurality of separate systems interface with an input/displaymanager. These systems can include, for example, an ultrasound system, areal time imaging system, a 3D mapping system, a Hemo system, an ECGsystem, and a 3D workstation. The input/display manager directs which ofthe displays of the various systems get displayed on the compositesystem, and manages their size and location, all in accordance with theworkflow, automatically updating the composite display each to theconfiguration appropriate for each step in the work flow.

Regardless of the configuration, the user can control any of the systemsdisplayed simply by moving the cursor over the appropriate portion ofthe composite display, and operating the graphical user interface of thesystem with the mouse and keyboard.

The components of the system 200 are shown schematically in FIG. 16,comprising a composite display 204, keyboard 206 and mouse 208, anddedicated controller 210. The connector plate for the input displaymanager is indicated as 230 in FIG. 16. The connector plate has a serialport 232 for connecting to the composite monitor 204 and a serial port234 for connecting to the video card of the computer running the system200, and a USB connector 236 for connecting to a USB port on thecomputer running the system 200. In addition the connector plate has amonitor, video card, and USB connections 238, 240, and 242 for theseparate navigation system; monitor, video card, and USB connections244, 246, and 248 for the separate live ECG system; monitor, video card,and USB connections 250, 252, and 254 for the separate reference ECGsystem; monitor, video card, and USB connections 256, 258, and 260 forthe separate x-ray imaging system; monitor, video card, and USBconnections 262, 264, and 266 mapping/localization system; monitor,video card, and USB connections 268, 270, and 272 for the separateultrasound system; two sets of monitor, video card, and USB connections274 A & B, 276 A & B, and 278 A & B for additional separate computerbased medical systems.

This arrangement allows the user to connect separate systems togetherwithout modifying the individual systems, and control them from thecomposite display 204 using the keyboard 206 and mouse 208. The system200 operates so that the mouse appears to move continuously across thecomposite display 204 across the portions that represent the displays ofthe individual systems, where the cursor is actually a cursor moving onthe interface of the separate system, and across the portions of thecomposite display between those portions, where the cursor is the cursorof the interface of the system 200. By managing the cursor in this way,it appears to be a single cursor moving continuously over the display200, even though it may actually be several cursors.

As shown in FIG. 18, when the cursor appears in a particular portion ofthe screen, the keyboard 206 and mouse 208 operate to control theseparate system in whose display the mouse is visible. Thus in FIG. 18A,the cursor displayed is the cursor in the Carto system, and operatingthe keyboard 208 operates the Carto system. In FIG. 18B, the cursor hasbeen moved to the portion of the composite display showing the displayof the ECG system. This is shown by the solid white cursor arrow, thefaded cursor arrows simply showing previous positions of the cursor,illustrating that the movement appears to the user to be continuous,even though in the preferred embodiment it is not continuous (andinstead is a series of the separate system cursor arrows bridged by thecursor arrow from the system). In FIG. 18B, since the cursor arrow is inthe ECG portion of the display, the keyboard 208 operates in the ECGsystem.

One embodiment of a seamless multi-system keyboard and mouse controller,referred to as a consolidated controller 400, uses a USB implementation.Consolidated controller 400 enables a single keyboard 412 and/or asingle mouse 414 to control the position of composite cursor 205 amongthe multiple display portions of consolidate display 408. Consolidatedcontroller 400 can also be characterized as a seamless cursor controlsystem and a seamless keyboard control system as it can seamlesslyinterface a single composite cursor and a single keyboard with multiplecomputer systems. FIG. 19-A shows a general system diagram for such anembodiment. Master Control Computer 402 receives display inputs 404 froma set of controlled medical systems or computers 401, 403, . . , 407,and generates a composite display output 406 that is sent to at leastone master screen(s) 408. Master Control Computer 402 also providesinputs to electronic controller 410, also referred to as a cursorcontrol device 410, that redirects command inputs such as test inputsfrom keyboard 412 to a selected controlled medical system or computerbased on current cursor position as determined at least in part byposition and movement of mouse 414. Other input signals to electroniccontroller 410 include mouse clicks and other inputs dependent uponspecific user interface (UIF) configuration(s). FIG. 19-B illustratesdetails of the electronic circuit/cursor control device 410 that is partof, or associated with, the master medical system or master controlcomputer 402 in one example of such an embodiment. Specifically, FIG.l9B shows a system control CPU of the control system that communicatesUSB differential data signals D+, D−associated with an input (from mouse414. or keyboard 412, for example) to a CY7C63823-PXC (USB peripheralcontroller) associated with the control system. The CY7C63823-PXC (USBperipheral controller) associated with the control system is inconnection with a “SPI” (Serial Peripheral Interface) bus, which isconnected to a CY7C63823-PXC (USB peripheral controller) for eachcontrolled system. The CY7C63823-PXC (USB peripheral controller)associated with the control system may communicate a SSEL (systemsession select) signal to a select CY7C63823-PXC 504associated with aselect controlled system, to thereby enable communication, via the SPIbus to a select controlled system within the set of controlled systems,of USB differential data signals D+, D− output to the controlled system.In this embodiment, the control system consists of electronic circuitsthat send appropriate keyboard and mouse events to a controlled systemmaking that system appear to function as one selectable window amongmany other similarly controlled systems. The electronic control systemutilizes standard computer interfaces including Universal Serial Bus(USB) to interface the keyboard and mouse control in such a way that thecontrolled system requires no specialized hardware or software to enablesuch control. The seamless keyboard and mouse controller 400 consists ofa master computer 402 and consolidated display 408 a control program onthe master computer system to arbitrate control events between multiplecontrolled systems, an electronic circuit 500 for simulating keyboardand mouse interfaces on controlled systems, and firmware programming totranslate master control events into standard keyboard and mouse eventsfor the controlled system. The firmware on the electronic controllerutilizes a standard Human Interface Descriptor (HID) to allow thesimulated mouse and keyboard events to be implemented with no hardwareor software modifications of the controlled system. As the operator usesthe master control computer 402 the control software determines whichcontrolled system should be subject to the operator's inputs. Thecontrol software then instructs the USB micro-controller 504corresponding to the selected controlled system(s) to generate simulatedkeyboard and mouse events directly interpretable by the controlledsystem, such that controlled system(s) appears to seamlessly functionwithin the context of the master system. The control system further usesa mouse control mode that allows the specific placement of the localcursor location on a controlled medical system to be specified by themaster system thus allowing the seamless motion of a cursor across manycomputer systems.

Thus embodiments of this facilitate the control of multiple medicalsystems with a single, simple interface system.

The consolidated system as described above limits the number ofsimultaneous keyboard and mouse controls to one. It is possible toretain the native keyboard and mouse of each system connected to theconsolidated system; however, these components do not operate similarlyto the consolidated system global keyboard and mouse. Instead, thesecomponents are limited to control the native system they are directlyconnected to rather than being capable of operating any system connectedto the consolidated system. In another embodiment of this invention, aconsolidated user interface is disclosed that controls a multiplicity ofmedical systems or computers, and a plurality of keyboard and micecombinations can be connected to the consolidated system to support atleast two simultaneous controls operating seamlessly with theconsolidated display.

In a first embodiment, two or more input devices combinations such askeyboard and mouse are connected to the consolidated system. Eachkeyboard may initially be set to focus its input on a separate systemcontrolled by the consolidated display (subject to the number of inputdevice combinations being not more than the number of controlledsystems). Each keyboard is associated with the system display it'scontrolling through a graphical indication; one example of such agraphical indication is a colored border around the window and acorresponding colored marker on the keyboard, or alternatively akeyboard of the corresponding color. Several design approaches enablesimultaneous control of at least one of a plurality of controlledsystems. In one embodiment, the position of each global cursor isindicated with a graphic such as a circle, halo, or similar graphicdifferentiator presented as an overlay constantly displayed over allwindows of the consolidated display. In this manner, as any of the miceis moved, the corresponding global cursor graphic is moved. In apreferred embodiment, the graphic does not block the central area of theglobal cursor position, so as to allow the native cursor of each systemto appear in its center. When the global cursor position of each mouseis located in a distinct associated system window, each mouse controlsits associated system independently without interference from any of theother mice; the native cursor of each system appearing in the center ofeach global cursor position graphic. When the global cursor positions oftwo or more mice are in the same system window, careful design enablesseamless operation without interference. For example, when a firstglobal cursor is in a first system window and a second global cursor isin a second system window, each global cursor begins with the nativecursor of the system in the center of the global cursor graphic. When afirst mouse is moved to translate a first global cursor to a secondsystem window, the native cursor for the second system disappears fromthe center of the second global cursor and immediately appears in thecenter of the first global cursor as soon as it crosses the border intothe second system window. In this implementation, the locus of controlswitches to the latest entrant into the window of the particular systemunder control. Effectively, only one mouse will control the system atany time even though the global cursor graphics of the two (or more)mice appear in the same window. A list of control priority is maintainedfor each control window, in such a manner that the first priority isalways with the latest entrant cursor, and prior cursor(s) that arestill in the window are pushed down the ordered list. Once a cursorleaves a given window, it cease to be listed in the priority list forthat window. Should the latest entrant cursor stop moving, the locus ofcontrol returns to the next cursor in the priority list that is moving,which then assumes the first priority in the priority list, and theformer “priority” cursor (the cursor having control) moves to the secondplace on the priority list. Should a second cursor start moving whilethe priority cursor is still moving, priority remains with the currentpriority cursor. If none of the cursors is moving, priority remains withthe last cursor having priority. Accordingly in such situations thenative cursor will always react to the currently moving mouse and in theevent two or more mice are concurrently moving, the mouse that initiatedmovement first maintains control of the native cursor. This priorityorder is shuffled only by a “new” cursor entering the window; or by thepriority cursor ceasing to move and an other cursor continuing orresuming motion. The global priority cursor graphic appears with amatching shape or color corresponding to the controlling mouse, and ishighlighted by use of a significantly higher intensity, larger shape, orsimilar differentiator that is not distracting to the users. With thisapproach, the operation appears nearly the same as having a consolidatedsystem mouse operating in the same window as a native mouse on the sameindependent system, but enables multi-user workflows from a single largedisplay. In addition, this embodiment of the present invention offers anew level of control for remote proctoring or procedure scenarios. Inthis manner, a remote proctor or physician can effectively work intandem with a local user of the system.

1. An integrated control and display system for operating a plurality of separate computer systems each having a visual display, comprising: a composite display configured to display the associated visual displays of at least two of the separate computer systems on the composite display; an input display system configured for connection to and receiving display input from each of the separate computer systems, and configured to output the associated visual displays to the composite display for displaying the associated visual displays of at least two of the separate computer systems; a single cursor control input device being movable for providing an input signal for controlling the position of a single cursor movable on the composite display among the visual displays of the separate computer systems; a master control computer in communication with the single cursor control input device and the input display system, being configured to generate the display of the visual displays associated with at least two of the separate computer systems on the composite display, and further being configured to assign the cursor control input device to one of the separate computer systems based on the position of the single cursor on the composite display, wherein the master control computer determines which separate computer system should be subject to inputs via the cursor control input device based on current cursor position as determined by position and movement of the single cursor control input device, so as to tie the single cursor control input device to a select separate computer system when the single cursor is positioned over the associated visual display of the select separate computer system, such that the single cursor control input device can be used to operate the computer system associated with the display on which the cursor appears; and a control circuit configured to connect the separate computer systems together without modifying the separate computer systems, comprising a serial bus connected to a plurality of Universal Serial Bus (USB) microcontrollers associated with each separate computer system, the control circuit being configured to receive inputs from the master control computer, and to translate said inputs into USB inputs, wherein the control circuit instructs the USB microcontroller corresponding to the select separate computer system to generate inputs directly interpretable by the select separate computer system, to thereby allow a user to manipulate the single cursor on the composite display using a single cursor control input device to control any of the separate computer systems whose visual displays are displayed on the composite display.
 2. The integrated control and display system according to claim 1 wherein the composite display is configured to display the visual displays of each of the separate computer systems, with the single cursor control input device functioning to move the cursor across the composite display and the visual displays of each of the separate computer systems displayed thereon.
 3. The integrated control and display system according to claim 2, further comprising a keyboard device connected to the master control computer, wherein when the single cursor is positioned over and clicked on the visual display of the same separate computer system, inputs to the keyboard device are input to the separate computer system for controlling the separate computer system.
 4. The integrated control and display system according to claim 2 wherein the system displays a composite cursor when the cursor is in the spaces between the visual displays of the separate computer systems, and displays a cursor of the separate computer system when the cursor is over the visual display of that separate computer system.
 5. The integrated control and display system according to claim 1 wherein each of the visual displays of the separate computer systems is displayed on a separate display window, and wherein the at least one cursor control device functions to move the cursor among the separate display windows.
 6. The integrated control and display system according to claim 5 wherein the displays for each of the visual displays are arranged in an array, and wherein the single cursor control input device functions to move the cursor from a display window to an immediately adjacent display window in the array.
 7. The integrated control and display system according to claim 1 wherein the cursor manipulated on the composite display changes in appearance on at least some of the visual displays of the separate computer systems.
 8. A method of operating a plurality of separate computer systems, each having a visual display, comprising: connecting separate computer systems to a control circuit that is configured to connect the separate computer systems together without modifying the separate computer systems, by connecting a serial bus to a plurality of Universal Serial Bus (USB) microcontrollers associated with each separate computer system; communicating display input from each of the separate computer systems via an input display system connected to each of the separate computer systems, which input display system is configured to communicate a video output including the associated visual displays; displaying the visual displays from each of a plurality of separate computer systems on a composite display; operating a single cursor control input device to manipulate a cursor to one of the visual displays associated with a separate computer system, and operating the single cursor control input device to operate the separate computer system corresponding to the visual display using a master control computer in communication with the single cursor control input device and the input display system, wherein the master control computer is configured to assign the cursor control input device to one of the separate computer systems based on the position of the single cursor on the composite display, and to determine which separate computer system should be subject to inputs via the cursor control input device based on current cursor position as determined by position and movement of the single cursor control input device, so as to tie the single cursor control input device to a select separate computer system when the single cursor is positioned over the associated visual display of the select separate computer system, such that the single cursor control input device can be used to operate the separate computer system by communicating USB inputs to the control circuit, which thereby instructs the USB microcontroller corresponding to the select separate computer system to generate inputs directly interpretably by the select separate computer system, to thereby allow a user to control any of the separate computer systems whose visual displays are displayed on the composite display.
 9. The method according to claim 8 wherein the visual displays for each of the separate computer systems are displayed on a composite display, and wherein the step of operating the single cursor control input device comprises operating the single cursor control input device to move a cursor on the composite display to a selected visual display, and operating the single cursor control input device to operate the separate computer system corresponding to the visual display.
 10. The method according to claim 9 wherein the cursor changes appearance on at least some of the visual displays of the separate computer systems.
 11. The method according to claim 9, further comprising the step of connecting a keyboard to the master control computer, wherein the keyboard input device can operate the separate computer system when the single cursor is positioned over and clicked on the same visual display corresponding to the separate computer system.
 12. The method according to claim 8 wherein the visual displays of the separate computer systems are displayed on separate displays, and wherein the step of operating the single cursor control input device comprises operating the single cursor control input device to move a cursor from display to display to a selected visual display, and operating the single cursor control input device to operate the separate computer system corresponding to the visual display.
 13. The method according to claim 12 wherein the cursor changes appearance on at least some of the visual displays of the separate computer systems.
 14. The method according to claim 12 wherein the displays for each of the visual displays are arranged in an array, and wherein the step of operating the single cursor control input device comprises operating the single cursor control input device to move a cursor from one display to an immediately adjacent display in the array.
 15. A control for operating a plurality of separate computer systems, the control comprising: a composite display configured to display the associated visual displays of at least two of the separate computer systems on the composite display; a video system having ports configured for connecting to and for receiving display input from each of the plurality of separate computer systems and displaying it on the composite display; a single cursor control input device being movable for providing an input signal for controlling the position of a single cursor movable on the composite display among the visual displays of the separate computer systems; a master control computer in communication with the single cursor control input device and the input display system, being configured to generate the display of the visual displays associated with at least two of the separate computer systems on the composite display, and further being configured to assign the cursor control input device to one of the separate computer systems based on the position of the single cursor on the composite display, wherein the master control computer determines which separate computer system should be subject to inputs via the cursor control input device based on current cursor position as determined by position and movement of the single cursor control input device, so as to tie the single cursor control input device to a select separate computer system when the single cursor is positioned over the associated visual display of the select separate computer system, such that the single cursor control input device can be used to operate the computer system associated with the display on which the single cursor appears; and a control circuit configured to connect separate computer systems together without modifying the separate computer systems, comprising a serial bus connected to a plurality of Universal Serial Bus (USB) microcontrollers associated with each separate computer system, the control circuit being configured to receive inputs from the master control computer and translate said inputs into USB inputs, wherein the control circuit instructs the USB microcontroller corresponding to the select separate computer system to generate inputs directly interpretable by the select separate computer system, to thereby allow a user to manipulate the single cursor on the composite display using a single cursor control input device to control any of the separate computer systems whose visual displays are displayed on the composite display.
 16. A control for operating a plurality of separate computer systems, the control comprising: a composite display configured to display the associated visual displays of at least two of the separate computer systems on the composite display; a video system having ports configured for connecting to and for receiving display input from each of the plurality of separate computer systems and displaying it on the composite display; a single cursor control input device being movable for providing an input signal for controlling the position of a single cursor movable on the composite display among the visual displays of the separate computer systems; a master control computer in communication with the single cursor control input device and the input display system, being configured to cause the display of the associated visual displays of at least two of the separate computer systems on the composite display, and further being configured to assign the cursor control input device to one of the separate computer systems based on the position of the single cursor on the composite display, wherein the master control computer determines which separate computer system should be subject to inputs via the cursor control input device based on current cursor position as determined by position and movement of the single cursor control input device, so as to tie the single cursor control input device to a select separate computer system when the single cursor is positioned over the associated visual display of the select separate computer system, such that the single cursor control input device can be used to operate the computer system associated with the display on which the single cursor appears; and a control circuit connected to the master control computer, and to each of the separate computer systems, being configured to connect the separate computer systems together without any hardware or software modifications of the separate computer systems, the control circuit comprising a serial bus connected to a plurality of Universal Serial Bus (USB) microcontrollers associated with each separate computer system, the control circuit being further configured to receive inputs from the master control computer and to translate said inputs into USB inputs, wherein the control circuit instructs the USB microcontroller corresponding to the select separate computer system to generate inputs directly interpretably by the select separate computer system, to thereby allow a user to manipulate the single cursor on the composite display using a single cursor control input device to control any of the separate computer systems whose visual displays are displayed on the composite display.
 17. A method of operating a plurality of separate computer systems comprising: communicating display input from each of the separate computer systems via an input display system connected to each of the separate computer systems, which input display system is configured to communicate a video output including the associated visual displays; displaying the separate visual displays from each of the plurality of separate computer systems on a composite display; connecting an interface computer with the single cursor control input device and the input display system; operating a single cursor control input device to manipulate a single cursor movable on the composite display to one of the visual displays associated with a separate computer system, wherein the interface computer is configured to assign the cursor control input device to one of the separate computer systems based on the position of the single cursor on the composite display, and to determine which separate computer system should be subject to inputs via the cursor control input device based on current cursor position as determined by position and movement of the single cursor control input device, so as to tie the single cursor control input device to a select separate computer system when the single cursor is positioned over the associated visual display of the select separate computer system; connecting the separate computer systems to a control circuit configured to connect the separate computer systems together without modifying the separate computer systems, by connecting a serial bus to a plurality of Universal Serial Bus (USB) microcontrollers associated with each separate computer system; and translating inputs generated by the operation of the single cursor control input device to control signals to an interface computer to move a cursor in the interstices between the separate displays on the composite screen and to control signals to the separate computer systems to operate the separate computer systems, so that the cursor appears to operate substantially continuously across the entire composite display, and when the cursor is positioned over the separate display of one of the separate computer systems, the single cursor control input device operates the separate computer system by communication of inputs as USB signals to the control circuit, which instructs the USB microcontroller corresponding to the select separate computer system to generate inputs directly interpretably by the select separate computer system.
 18. The method according to claim 17 wherein the step of translating inputs generated by the operation of the single cursor control input device to control signals to an interface computer and to control signals to the separate computer systems includes communicating the signals using the USB protocol.
 19. The method according to claim 18 wherein the relative movements of the single cursor control input device relative to the composite display are translated into absolute cursor positions communicated to the separate computer systems using the USB protocol.
 20. The method according to claim 17 wherein the step of translating inputs comprises scaling to provide smooth cursor movement, as the cursor moves across the composite display.
 21. A control for operating a plurality of separate computer systems, the control comprising: a video system configured for connection to and receiving display input from each of the separate computer systems, and configured to output the associated visual displays to the composite display for displaying the associated visual displays of at least two of the separate computer systems; a composite display including the visual displays from at least some of the plurality of separate computer systems; a single cursor control input device providing cursor control input for the composite display which is translated to a corresponding local cursor location on each of the separate computer systems when the cursor on the composite display is on the display corresponding to that separate computer system. a master control computer in communication with the single cursor control input device and the video system, being configured to generate the display of the visual displays from at least some of the plurality of separate computer systems on the composite display, and further being configured to assign the cursor control input device to one of the separate computer systems based on the position of the single cursor on the composite display, wherein the master control computer determines which separate computer system should be subject to inputs via the cursor control input device based on current cursor position as determined by position and movement of the single cursor control input device, so as to tie the single cursor control input device to a select separate computer system when the single cursor is positioned over the associated visual display of the select separate computer system, such that the single cursor control input device can be used to operate the select computer system associated with the display on which the single cursor appears; and a control circuit that is configured to connect the separate computer systems together without any hardware or software modifications of the separate computer systems, comprising a serial bus connected to a plurality of Universal Serial Bus (USB) microcontrollers associated with each separate computer system, the control circuit being configured to receive inputs from the master control computer and to translate said inputs into USB inputs, wherein the cursor control input provided by the single cursor control input device is translated by the master control computer and the control circuit to a corresponding cursor location associated with a select separate computer system when the single cursor on the composite display is on the visual display corresponding to the select separate computer system, to thereby allow a user to manipulate the single cursor on the composite display using a single cursor control input device to control any of the separate computer systems whose visual displays are displayed on the composite display.
 22. The control according to claim 21 further comprising a video system for receiving display input from each of the plurality of separate computer systems and displaying the visual displays for each of the separate computer systems on the composite display.
 23. An integrating control and display system for operating a plurality of separate computer systems, comprising: a composite display integrating the separate displays from each of a plurality of separate computer systems; an input display system having ports configured for connecting to and receiving display input from each of the separate computer systems, and configured to output the associated visual displays on the composite display; a single cursor control input device for manipulating a cursor on the composite display, including on the displays of each of the separate computer systems on the composite display, the cursor control device operating the system corresponding to the separate display on which it appears on the composite display; a master control computer in communication with the single cursor control input device and the input display system, being configured to generate the display of the visual displays from at least some of the plurality of separate computer systems on the composite display, and further being configured to assign the cursor control input device to one of the separate computer systems based on the position of the single cursor on the composite display, wherein the master control computer determines which separate computer system should be subject to inputs via the cursor control input device based on current cursor position as determined by position and movement of the single cursor control input device, so as to tie the single cursor control input device to a select separate computer system when the single cursor is positioned over the associated visual display of the select separate computer system, such that the single cursor control input device can be used to operate the select computer system associated with the display on which the single cursor appears; and a control circuit that is configured to connect the separate computer systems together without any hardware or software modifications of the separate computer systems, comprising a serial bus connected to a plurality of Universal Serial Bus (USB) microcontrollers associated with each separate computer system, the control circuit being configured to receive inputs from the master control computer and to translate said inputs into USB inputs, wherein the control circuit instructs the USB microcontroller corresponding to the select separate computer system to generate inputs directly interpretably by the select separate computer system, to thereby allow a user to manipulate the single cursor on the composite display using a single cursor control input device to control any of the separate computer systems whose visual displays are displayed on the composite display.
 24. A method of operating a plurality of separate computer systems without altering the hardware or software of the separate computer systems, the method comprising: connecting the plurality of separate computer systems to a control circuit that is configured to connect the separate computer systems together without any hardware or software modifications of the separate computer systems, by connecting a serial bus to a plurality of Universal Serial Bus (USB) microcontrollers associated with each separate computer system; connecting the plurality of separate computer systems to a video system to provide input from each of the separate computer systems to the video system, which is configured to communicate a video output including the associated visual displays of each of the plurality of separate computer systems; displaying the separate displays of each of the plurality of separate computer systems on a composite display of a composite computer system; operating the single cursor control input device of the composite computer system to manipulate a cursor on the composite display, and using a computer interface in communication with the single cursor control input device to assign the cursor control input device to one of the separate computer systems such that when the cursor on the composite display overlays the separate display of one of the separate computer systems, cursor control signals are generated to operate the separate computer system corresponding to the operation of the cursor control device of the composite system using the computer interface, wherein the computer interface is configured to assign the cursor control input device to one of the separate computer systems based on the position of the single cursor on the composite display, and to determine which separate computer system should be subject to inputs via the cursor control input device based on current cursor position as determined by position and movement of the single cursor control input device, so as to tie the single cursor control input device to a select separate computer system when the single cursor is positioned over the associated visual display of the select separate computer system; and wherein the single cursor control input device can be used to operate the separate computer system by communicating USB inputs to the control circuit, which thereby instructs the USB microcontroller corresponding to the select separate computer system to generate inputs directly interpretably by the select separate computer system, to thereby allow a user to control any of the separate computer systems whose visual displays are displayed on the composite display.
 25. The method of claim 24 wherein the cursor control signals are generated with a USB computer interface for operating the separate computer system.
 26. A method of controlling a plurality of separate computer systems, comprising: connecting the plurality of separate computer systems to a control circuit that is configured to connect the separate computer systems together without any hardware or software modifications of the separate computer systems, by connecting a serial bus to a plurality of Universal Serial Bus (USB) microcontrollers associated with each separate computer system; connecting the plurality of separate computer systems to a video system to provide input from each of the separate computer systems to the video system, which is configured to communicate a video output including the associated visual displays of each of the plurality of separate computer systems; displaying the display from each of the separate computer systems on an integrated display; accepting inputs from a single cursor control input device to move a cursor across the integrated display, the single cursor control input device being operable to operate the separate computer system over which the cursor is displayed on the integrated display; connecting a master control computer to the single cursor control input device and the input display system, wherein the master control computer is configured to assign the cursor control input device to one of the separate computer systems based on the position of the single cursor on the composite display, and to determine which separate computer system should be subject to inputs via the cursor control input device based on current cursor position as determined by position and movement of the single cursor control input device, so as to tie the single cursor control input device to a select separate computer system when the single cursor is positioned over the associated visual display of the select separate computer system; and wherein the cursor control inputs from the master control computer and translate said inputs into USB inputs that the master control computer provides to the control circuit, and the control circuit instructs the USS microcontroller corresponding to the select separate computer system to generate inputs directly interpretably by the select separate computer system, to thereby allow the user to manipulate a single cursor over a composite display using a single cursor control input device to control any of the separate computer systems whose visual displays are displayed on the composite display. 